{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6a51548b",
   "metadata": {},
   "source": [
    "# Deploy gpt-oss\n",
    "\n",
    "<div align=\"left\">\n",
    "<a target=\"_blank\" href=\"https://console.anyscale.com/template-preview/deployment-serve-llm?file=%252Ffiles%252Fgpt-oss\"><img src=\"https://img.shields.io/badge/🚀 Run_on-Anyscale-9hf\"></a>&nbsp;\n",
    "<a href=\"https://github.com/ray-project/ray/tree/master/doc/source/serve/tutorials/deployment-serve-llm/gpt-oss\" role=\"button\"><img src=\"https://img.shields.io/static/v1?label=&amp;message=View%20On%20GitHub&amp;color=586069&amp;logo=github&amp;labelColor=2f363d\"></a>&nbsp;\n",
    "</div>\n",
    "\n",
    "*gpt-oss* is a family of open-source models designed for general-purpose language understanding and generation. The 20&nbsp;B parameter variant (`gpt-oss-20b`) offers strong reasoning capabilities with lower latency. This makes it well-suited for local or specialized use cases. The larger 120&nbsp;B parameter variant (`gpt-oss-120b`) is designed for production-scale, high-reasoning workloads.\n",
    "\n",
    "For more information, see the [gpt-oss collection](https://huggingface.co/collections/openai/gpt-oss-68911959590a1634ba11c7a4).\n",
    "\n",
    "---\n",
    "\n",
    "## Configure Ray Serve LLM\n",
    "\n",
    "Ray Serve LLM provides multiple [Python APIs](https://docs.ray.io/en/latest/serve/api/index.html#llm-api) for defining your application. Use [`build_openai_app`](https://docs.ray.io/en/latest/serve/api/doc/ray.serve.llm.build_openai_app.html#ray.serve.llm.build_openai_app) to build a full application from your [`LLMConfig`](https://docs.ray.io/en/latest/serve/api/doc/ray.serve.llm.LLMConfig.html#ray.serve.llm.LLMConfig) object.\n",
    "\n",
    "Below are example configurations for both gpt-oss-20b and gpt-oss-120b, depending on your hardware and use case.\n",
    "\n",
    "---\n",
    "\n",
    "### gpt-oss-20b\n",
    "\n",
    "To deploy a small-sized model such as gpt-oss-20b, a single GPU is sufficient:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86070ffe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# serve_gpt_oss.py\n",
    "from ray.serve.llm import LLMConfig, build_openai_app\n",
    "\n",
    "llm_config = LLMConfig(\n",
    "    model_loading_config=dict(\n",
    "        model_id=\"my-gpt-oss\",\n",
    "        model_source=\"openai/gpt-oss-20b\",\n",
    "    ),\n",
    "    accelerator_type=\"L4\",\n",
    "    deployment_config=dict(\n",
    "        autoscaling_config=dict(\n",
    "            min_replicas=1,\n",
    "            max_replicas=2,\n",
    "        )\n",
    "    ),\n",
    "    engine_kwargs=dict(\n",
    "        max_model_len=32768\n",
    "    ),\n",
    ")\n",
    "\n",
    "app = build_openai_app({\"llm_configs\": [llm_config]})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adeb0b16",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "\n",
    "### gpt-oss-120b\n",
    "\n",
    "To deploy a medium-sized model such as `gpt-oss-120b`, a single node with multiple GPUs is sufficient. Set `tensor_parallel_size` to distribute the model’s weights across the GPUs in your instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ac648e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# serve_gpt_oss.py\n",
    "from ray.serve.llm import LLMConfig, build_openai_app\n",
    "\n",
    "llm_config = LLMConfig(\n",
    "    model_loading_config=dict(\n",
    "        model_id=\"my-gpt-oss\",\n",
    "        model_source=\"openai/gpt-oss-120b\",\n",
    "    ),\n",
    "    accelerator_type=\"L40S\", # Or \"A100-40G\"\n",
    "    deployment_config=dict(\n",
    "        autoscaling_config=dict(\n",
    "            min_replicas=1,\n",
    "            max_replicas=2,\n",
    "        )\n",
    "    ),\n",
    "    engine_kwargs=dict(\n",
    "        max_model_len=32768,\n",
    "        tensor_parallel_size=2,\n",
    "    ),\n",
    ")\n",
    "\n",
    "app = build_openai_app({\"llm_configs\": [llm_config]})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b17a7140",
   "metadata": {},
   "source": [
    "**Note:** Before moving to a production setup, migrate to using a [Serve config file](https://docs.ray.io/en/latest/serve/production-guide/config.html) to make your deployment version-controlled, reproducible, and easier to maintain for CI/CD pipelines. For an example, see [Serving LLMs - Quickstart Examples: Production Guide](https://docs.ray.io/en/latest/serve/llm/quick-start.html#production-deployment).\n",
    "\n",
    "---\n",
    "\n",
    "## Deploy locally\n",
    "\n",
    "### Prerequisites\n",
    "\n",
    "* Access to GPU compute.\n",
    "\n",
    "### Dependencies\n",
    "\n",
    "gpt-oss integration is available starting from `ray>=2.49.0` and `vllm==0.10.1`.\n",
    "\n",
    "```bash\n",
    "pip install \"ray[serve,llm]>=2.49.0\"\n",
    "pip install \"vllm==0.10.1\"\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "### Launch the service\n",
    "\n",
    "Follow the instructions in [Configure Ray Serve LLM](#configure-ray-serve-llm) according to the model size you choose, and define your app in a Python module `serve_gpt_oss.py`.\n",
    "\n",
    "In a terminal, run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbdb0921",
   "metadata": {},
   "outputs": [],
   "source": [
    "serve run serve_gpt_oss:app --non-blocking"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df944967",
   "metadata": {},
   "source": [
    "Deployment typically takes a few minutes as Ray provisions the cluster, the vLLM server starts, and Ray Serve downloads the model.\n",
    "\n",
    "---\n",
    "\n",
    "### Send requests\n",
    "\n",
    "Your endpoint is available locally at `http://localhost:8000`. You can use a placeholder authentication token for the OpenAI client, for example `\"FAKE_KEY\"`.\n",
    "\n",
    "#### Example curl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5309437",
   "metadata": {},
   "outputs": [],
   "source": [
    "curl -X POST http://localhost:8000/v1/chat/completions \\\n",
    "  -H \"Authorization: Bearer FAKE_KEY\" \\\n",
    "  -H \"Content-Type: application/json\" \\\n",
    "  -d '{ \"model\": \"my-gpt-oss\", \"messages\": [{\"role\": \"user\", \"content\": \"How many Rs in strawberry ?\"}] }'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d623a30f",
   "metadata": {},
   "source": [
    "#### Example Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75bedc22",
   "metadata": {},
   "outputs": [],
   "source": [
    "#client.py\n",
    "from urllib.parse import urljoin\n",
    "from openai import OpenAI\n",
    "\n",
    "api_key = \"FAKE_KEY\"\n",
    "base_url = \"http://localhost:8000\"\n",
    "\n",
    "client = OpenAI(base_url=urljoin(base_url, \"v1\"), api_key=api_key)\n",
    "\n",
    "# Example query\n",
    "response = client.chat.completions.create(\n",
    "    model=\"my-gpt-oss\",\n",
    "    messages=[\n",
    "        {\"role\": \"user\", \"content\": \"How many r's in strawberry\"}\n",
    "    ],\n",
    "    stream=True\n",
    ")\n",
    "\n",
    "# Stream\n",
    "for chunk in response:\n",
    "    # Stream reasoning content\n",
    "    if hasattr(chunk.choices[0].delta, \"reasoning_content\"):\n",
    "        data_reasoning = chunk.choices[0].delta.reasoning_content\n",
    "        if data_reasoning:\n",
    "            print(data_reasoning, end=\"\", flush=True)\n",
    "    # Later, stream the final answer\n",
    "    if hasattr(chunk.choices[0].delta, \"content\"):\n",
    "        data_content = chunk.choices[0].delta.content\n",
    "        if data_content:\n",
    "            print(data_content, end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b095ebf3",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "\n",
    "### Shut down the service\n",
    "\n",
    "To shutdown your LLM service: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fd3dacf",
   "metadata": {},
   "outputs": [],
   "source": [
    "serve shutdown -y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb81fa41",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "\n",
    "## Deploy to production with Anyscale services\n",
    "\n",
    "For production deployment, use Anyscale services to deploy the Ray Serve app to a dedicated cluster without modifying the code. Anyscale ensures scalability, fault tolerance, and load balancing, keeping the service resilient against node failures, high traffic, and rolling updates.\n",
    "\n",
    "---\n",
    "\n",
    "### Launch the service\n",
    "\n",
    "Anyscale provides out-of-the-box images (`anyscale/ray-llm`), which come pre-loaded with Ray Serve LLM, vLLM, and all required GPU and runtime dependencies. See the [Anyscale base images](https://docs.anyscale.com/reference/base-images) for details on what each image includes.\n",
    "\n",
    "Build a minimal Dockerfile:\n",
    "```Dockerfile\n",
    "FROM anyscale/ray:2.49.0-slim-py312-cu128\n",
    "\n",
    "# C compiler for Triton’s runtime build step (vLLM V1 engine)\n",
    "# https://github.com/vllm-project/vllm/issues/2997\n",
    "RUN sudo apt-get update && \\\n",
    "    sudo apt-get install -y --no-install-recommends build-essential\n",
    "\n",
    "RUN pip install vllm==0.10.1\n",
    "```\n",
    "\n",
    "Create your Anyscale service configuration in a new `service.yaml` file and reference the Dockerfile with `containerfile`:\n",
    "\n",
    "```yaml\n",
    "# service.yaml\n",
    "name: deploy-gpt-oss\n",
    "containerfile: ./Dockerfile # Build Ray Serve LLM with vllm==0.10.1\n",
    "compute_config:\n",
    "  auto_select_worker_config: true \n",
    "working_dir: .\n",
    "cloud:\n",
    "applications:\n",
    "  # Point to your app in your Python module\n",
    "  - import_path: serve_gpt_oss:app\n",
    "```\n",
    "\n",
    "\n",
    "Deploy your service:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fa0556b",
   "metadata": {},
   "outputs": [],
   "source": [
    "anyscale service deploy -f service.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e6de36c",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "\n",
    "### Send requests \n",
    "\n",
    "The `anyscale service deploy` command output shows both the endpoint and authentication token:\n",
    "\n",
    "```console\n",
    "(anyscale +3.9s) curl -H \"Authorization: Bearer <YOUR-TOKEN>\" <YOUR-ENDPOINT>\n",
    "```\n",
    "\n",
    "You can also retrieve both from the service page in the Anyscale console. Click **Query** at the top. See [Send requests](#send-requests) for example requests, but make sure to use the correct endpoint and authentication token.  \n",
    "\n",
    "---\n",
    "\n",
    "### Access the Serve LLM dashboard\n",
    "\n",
    "For instructions on enabling LLM-specific logging, see [Enable LLM monitoring](#enable-llm-monitoring). To open the Ray Serve LLM Dashboard from an Anyscale service:\n",
    "\n",
    "1. In the Anyscale console, go to the **Service** or **Workspace** tab.\n",
    "1. Navigate to the **Metrics** tab.\n",
    "1. Click **View in Grafana** and click **Serve LLM Dashboard**.\n",
    "\n",
    "---\n",
    "\n",
    "### Shutdown\n",
    "\n",
    "To shutdown your Anyscale Service:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "474b2764",
   "metadata": {},
   "outputs": [],
   "source": [
    "anyscale service terminate -n deploy-gpt-oss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49f67c39",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "\n",
    "## Enable LLM monitoring\n",
    "\n",
    "The *Serve LLM Dashboard* offers deep visibility into model performance, latency, and system behavior, including:\n",
    "\n",
    "- Token throughput (tokens/sec).\n",
    "- Latency metrics: Time To First Token (TTFT), Time Per Output Token (TPOT).\n",
    "- KV cache utilization.\n",
    "\n",
    "To enable these metrics, go to your LLM config and set `log_engine_metrics: true`:\n",
    "\n",
    "```yaml\n",
    "applications:\n",
    "- ...\n",
    "  args:\n",
    "    llm_configs:\n",
    "      - ...\n",
    "        log_engine_metrics: true\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Improve concurrency\n",
    "\n",
    "Ray Serve LLM uses [vLLM](https://docs.vllm.ai/en/stable/) as its backend engine, which logs the *maximum concurrency* it can support based on your configuration.\n",
    "\n",
    "Example log for gpt-oss-20b with 1xL4:\n",
    "```console\n",
    "INFO 09-08 17:34:28 [kv_cache_utils.py:1017] Maximum concurrency for 32,768 tokens per request: 5.22x\n",
    "```\n",
    "\n",
    "Example log for gpt-oss-120b with 2xL40S:\n",
    "```console\n",
    "INFO 09-09 00:32:32 [kv_cache_utils.py:1017] Maximum concurrency for 32,768 tokens per request: 6.18x\n",
    "```\n",
    "\n",
    "To improve concurrency for gpt-oss models, see [Deploy a small-sized LLM: Improve concurrency](https://docs.ray.io/en/latest/serve/tutorials/deployment-serve-llm/small-size-llm/README.html#improve-concurrency) for small-sized models such as `gpt-oss-20b`, and [Deploy a medium-sized LLM: Improve concurrency](https://docs.ray.io/en/latest/serve/tutorials/deployment-serve-llm/medium-size-llm/README.html#improve-concurrency) for medium-sized models such as `gpt-oss-120b`.\n",
    "\n",
    "**Note:** Some example guides recommend using quantization to boost concurrency. `gpt-oss` weights are already 4-bit by default, so further quantization typically isn’t applicable.  \n",
    "\n",
    "For broader guidance, also see [Choose a GPU for LLM serving](https://docs.anyscale.com/llm/serving/gpu-guidance) and [Optimize performance for Ray Serve LLM](https://docs.anyscale.com/llm/serving/performance-optimization).\n",
    "\n",
    "---\n",
    "\n",
    "## Reasoning configuration\n",
    "\n",
    "You don’t need a custom reasoning parser when deploying `gpt-oss` with Ray Serve LLM, you can access the reasoning content in the model's response directly. You can also control the reasoning effort of the model in the request.\n",
    "\n",
    "---\n",
    "\n",
    "### Access reasoning output\n",
    "\n",
    "The reasoning content is available directly in the `reasoning_content` field of the response:\n",
    "\n",
    "```python\n",
    "response = client.chat.completions.create(\n",
    "    model=\"my-gpt-oss\",\n",
    "    messages=[\n",
    "        ...\n",
    "    ]\n",
    ")\n",
    "reasoning_content = response.choices[0].message.reasoning_content\n",
    "content = response.choices[0].message.content\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "### Control reasoning effort\n",
    "\n",
    "`gpt-oss` supports [three reasoning levels](https://huggingface.co/openai/gpt-oss-20b#reasoning-levels): **low**, **medium**, and **high**. The default level is **medium**.\n",
    "\n",
    "You can control reasoning with the `reasoning_effort` request parameter:  \n",
    "```python\n",
    "response = client.chat.completions.create(\n",
    "    model=\"my-gpt-oss\",\n",
    "    messages=[\n",
    "        {\"role\": \"user\", \"content\": \"What are the three main touristic spots to see in Paris?\"}\n",
    "    ],\n",
    "    reasoning_effort=\"low\" # Or \"medium\", \"high\"\n",
    ")\n",
    "```\n",
    "\n",
    "You can also set a level explicitly in the system prompt:  \n",
    "```python\n",
    "response = client.chat.completions.create(\n",
    "    model=\"my-gpt-oss\",\n",
    "    messages=[\n",
    "        {\"role\": \"system\", \"content\": \"Reasoning: low. You are an AI travel assistant.\"},\n",
    "        {\"role\": \"user\", \"content\": \"What are the three main touristic spots to see in Paris?\"}\n",
    "    ]\n",
    ")\n",
    "```\n",
    "\n",
    "**Note:** There's no reliable way to completely disable reasoning.\n",
    "\n",
    "---\n",
    "\n",
    "## Troubleshooting\n",
    "\n",
    "### Can't download the vocab file  \n",
    "```console\n",
    "openai_harmony.HarmonyError: error downloading or loading vocab file: failed to download or load vocab\n",
    "```\n",
    "\n",
    "The `openai_harmony` library needs the *tiktoken* encoding files and tries to fetch them from OpenAI's public host. Common causes include:\n",
    "- Corporate firewall or proxy blocks `openaipublic.blob.core.windows.net`. You may need to whitelist this domain.\n",
    "- Intermittent network issues.\n",
    "- Race conditions when multiple processes try to download to the same cache. This can happen when [deploying multiple models at the same time](https://github.com/openai/harmony/pull/41).\n",
    "\n",
    "You can also directly download the *tiktoken* encoding files in advance and set the `TIKTOKEN_ENCODINGS_BASE` environment variable:\n",
    "```bash\n",
    "mkdir -p tiktoken_encodings\n",
    "wget -O tiktoken_encodings/o200k_base.tiktoken \"https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken\"\n",
    "wget -O tiktoken_encodings/cl100k_base.tiktoken \"https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken\"\n",
    "export TIKTOKEN_ENCODINGS_BASE=${PWD}/tiktoken_encodings\n",
    "```\n",
    "\n",
    "### `gpt-oss` architecture not recognized \n",
    "```console\n",
    "Value error, The checkpoint you are trying to load has model type `gpt_oss` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.\n",
    "```\n",
    "Older vLLM and Transformers versions don't register `gpt_oss`, raising an error when vLLM hands off to Transformers. Upgrade **vLLM ≥ 0.10.1** and let your package resolver such as `pip` handle the other dependencies.\n",
    "```bash\n",
    "pip install -U \"vllm>=0.10.1\"\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "In this tutorial, you learned how to deploy `gpt-oss` models with Ray Serve LLM, from development to production. You learned how to configure Ray Serve LLM, deploy your service on a Ray cluster, send requests, and monitor your service."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "repo_ray_docs",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.12.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
